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Abstract 


Ballistic  test  and  evaluation  of  body  armor  is  of  great  priority  for  the  US  Army  to  meet 
current  operational  needs.  The  current  experimental  design  process  centers  on  standardized  threat 
level  classifications  and  a  baseline  ballistic  limit  velocity,  V50;  both  measures  originate  from  the 
National  Institute  of  Justice  circa  1979.  The  measures  are  complex,  statistical  in  nature,  and  yield 
large  quantities  of  data.  A  methodology  incorporating  response  surface  techniques  improves 
ballistic  test  and  evaluation  from  a  pass  fail  analysis  of  data  to  iterative,  directional  experiments 
with  design  intelligence.  Without  this  mathematical  direction,  it  is  extremely  difficult  to  analyze 
the  multitude  of  factors  and  their  interaction  effects  in  order  to  attain  product  improvement.  We 
provide  a  ballistic  experimental  design  example  to  demonstrate  the  usefulness  of  this 
methodology  and  identify  the  potential  for  its  application  in  future  armor  developments. 


iii 


About  the  Author(s) 

Cadet  Shane  Sullivan  is  a  senior  undergraduate  student  at  the  United  States  Military 
Academy  at  West  Point.  He  is  majoring  in  Engineering  Management  with  Electrical 
Engineering  focus  in  the  Department  of  Systems  Engineering,  and  he  will  be  commissioned 
as  an  aviation  officer  in  the  United  States  Army  in  May  of  2006. 


Acknowledgements 

This  project  incorporates  the  expert  knowledge  of  scientist  Dr.  James  Zheng,  Program 
Director  of  Soldier  Equipment  for  PEO  Soldier.  His  knowledge  the  experimental  design 
process  assisted  in  the  development  of  this  methodology.  Additionally,  LTC  John  Halstead, 
Ph.D.  provided  the  mathematical  knowledge  and  digital  code  for  the  Response  Surface 
portion  of  this  project. 


IV 


Table  of  Contents 


Abstract . iii 

About  the  Author(s) . iv 

Acknowledgements . iv 

Table  of  Contents . v 

List  of  Figures . vi 

List  of  Tables . vi 

Chapter  1:  Introduction . 1 

Chapter  2:  Proposed  Methodology . 1 

2.1  Response  Surface  Techniques . 1 

2.2  Iterative  Process . 3 

Chapter  3 :  Applied  Experiment . 4 

3.1  Purpose . 4 

3.2  Factor  Design . 5 

3.3  Response . 5 

Chapter  4:  Results  of  the  Experiment . 6 

4. 1  Significant  Factors . 6 

4.2  Response  Surface . 7 

4.3  Analysis  and  Application  of  the  Response  Surface . 8 

Chapter  5:  Conclusion . 9 

Bibliography . 10 

Appendix  A:  List  of  Abbreviations . 11 

Distribution  List . 12 


Chapter  6:  REPORT  DOCUMENTATION  PAGE  -  SF298 


13 


List  of  Figures 

Figure  1:  The  Iterative  Nature  of  Experimentation 
Figure  2:  Statistical  Ballistic  Response 
Figure  3:  Normal  Plot  of  Effects 
Figure  3:  Response  Surface  of  Body  Armor 


List  of  Tables 

Table  1,  Factor  Levels 
Table  2,  Experimental  Design 
Table  3,  Applied  OTV  Experiment 
Table  4,  Analysis  of  Variance 


VI 


Chapter  1:  Introduction 

Current  experimental  design  methodology  utilized  by  Army  researchers  and  armor 
industry  material  scientists  primarily  determines  the  effectiveness  of  a  given  armor  desiga  The 
scientists’  mission  is  to  develop  body  armor  systems  that  are  effective  in  defeating  multiple 
threats  under  various  environmental  conditions,  minimize  weight,  maximize  comfort  and  utility, 
and  perform  consistently  within  published  specifications.  Scientists  must  consider  a  myriad  of 
factors  including  material,  areal  density,  composition,  weave,  resins,  production  methods,  and  a 
continuously  growing  list  of  others.  In  addition,  they  must  measure  performance  in  meeting 
specifications  for  standard  threat  level  classifications,  pass-fail  penetration  for  multiple  round 
types,  statically  based  velocity  thresholds  for  multiple  round  types,  and  other  vital  experimental 
responses.  This  complex  experimental  environment  requires  both  expert  knowledge  and  a  large 
number  of  design  points,  especially  to  maintain  accuracy  in  evaluations  for  protecting  human 
life. 

Decision  makers  receive  data  from  multiple  ballistic  laboratories,  organized  in  batches 
that  cause  analytical  comparisons  and  multivariable  analysis  particularly  troublesome.  The 
decision  maker  relies  on  expert  knowledge  and  intimate  familiarization  with  the  data  to  make 
recommendations  for  further  system  enhancement  or  future  directional  design  iterations. 

A  change  in  current  methodology  would  offer  the  decision  maker  intelligent  design  by 
providing  surface  response,  factor  reduction,  and  an  easily  adaptable,  flexible  process  to 
evaluate,  analyze,  and  improve  armor  systems  design. 


Chapter  2:  Proposed  Methodology 


2.1  Response  Surface  Techniques 

Response  Surface  Methodology  (RSM)  is  documented  by  Myers  and  Montgomery  as  “a 
collection  of  statistical  and  mathematical  techniques  useful  for  developing,  improving,  and 
optimizing  processes.”  The  experimental  design  can  include  any  number  of  factors  and 
responses.  The  factors  are  encoded  to  achieve  orthogonal  design  using  the  transformation 


equation 
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The  result  of  this  transformation  is  the 
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E\  y]  =  /* f  XX  X  ) 

functional  approximation  LXJ  J  v  1  ’  2’  3‘"’  A  preliminary  experiment,  called  a 
screening  experiment,  investigates  the  statistical  importance  of  these  factors  in  hopes  of  reducing 
the  design  by  eliminating  the  unimportant  ones.  Table  1  is  a  design  example  of  the  natural 
variables  that  would  result  from  a  preliminary  experiment,  including  in-significant  factors. 

TABLE  I 
Factor  Levels 
Level 


Factors  Low  (-)  High  (+) 

Factor  A  Value  Value 

Factor  B  Value  Value 

Factor  C  Value  Value 


The  actual  experimental  design  includes  all  necessary  factor  combinations,  including 
interactions,  much  like  the  example  in  the  Table  2. 

TABLE  2 


Experimental  Design 


Design 

Factor  A 

Factor  B 

Factor  C 

Response 

0) 

-1 

-1 

-1 

Value  /  Binomial 

A 

1 

-1 

-1 

B 

-1 

1 

-1 

AB 

1 

1 

-1 

C 

-1  ■ 

-1 

1 

AC 

1 

-1 

] 

BC 

-1 

1 

] 

f  . 

ABC 

1 

1 

1 

The  experiment  is  orthogonal  in  design,  providing  powerful  least  squares  regressions  within  a 
small  order  region  of  experimentation  in  a  given  multivariable  space.  The  standard  form  logistic 


regression  function  is  ^  +^ixi  +  ^2X2  +---+hxk  &  x _  jn  order  to  achieve  this  function 

with  the  orthogonal  design  used  in  this  methodology,  we  first  have  to  achieve  linear  regression 
through  log  odds.  This  logit  function  shows  the  probability  x=l  in  a  given  distribution 
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1  +  exp  r  ^  and  the  probability  x  =  0  is 


O(x) 
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exp 


.  This 
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summarizes  into  one  equation  for  the  odds  of  an  event  x  given  as  1  n  (x) 

Taking  the  natural  log  of  these  odds  and  you  return  to  the  standard  logistic  regression  of  beta 

matrix  transposed  x  shown  as  ]=PX  when  the  design  matrix  requires  the  use 

of  a  binomial  response,  such  as  whether  or  not  a  round  penetrated  an  armor  material  or  simply 
met  a  given  standard,  the  design  requires  a  mathematical  model  known  as  a  logit  link.  In  RSM 
the  logistic  regression  function  is  essentially  attempting  to  achieve  a  mean,  or  expected,  value. 


The  relationship  is  shown  mathematically  as  Bx)  4  + 4*i +  4*2  +  ■  •  • + hxk  jf  this 

is  true,  then  E\_y]  —  B  X  gecause  we  know  in  a  binomial  response  ^  e  ,  we  know 
values  cannot  exist  between  y  and  0.  This  requires  we  transpose  with  the  logit  link 

ln£fy]  =\n/u=B1x^>fi-eBlx  ^>odds^>\n\B—]  -BTx 

dl  .  Even  with  a  binomial  response,  the  mathematics 

return  to  the  original  logistic  regression  of  beta  matrix  transposed  x  with  this  logit  link. 

The  benefit  of  linear  regression  is  that  reduces  design  points  and  allows  for  multifactor 

design  through  an  iterative  process  that  searches  for  a  near  optimum  using  the  method  of  steepest 

ascent  or  decent.  If  multiple  responses  exist,  the  scientist  can  superimpose  response  surface 

contours  into  more  complex  process  optimization  problem  or  develop  a  binomial  response  from 

standards  based  performance  specifications  that  will  utilize  the  logit  link  model  shown  above. 

Special  consideration  in  the  experimental  design  can  also  reduce  variability  from  noise  factors 

through  robust  parameter  design. 

2.2  Iterative  Process 

The  iterative  process  this  methodology  incorporates  with  RSM  includes  a  repetitive 
sequence  of  conjecture,  design,  experiment,  and  analysis.  This  sequence  is  only  constrained  by 
the  conjecture  and  the  experiment.  Additionally,  the  sequence  does  not  move  linearly,  meaning 
conjuring  can  begin  prior  to  completion  of  the  first  phase  of  analysis.  The  subsequent  design  will 
incorporate  the  information  of  the  first  along  with  the  conjecture.  This  cycle  repeats  as  long  as 
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necessary  to  reach  the  objective  of  the  design  process.  Box  and  Draper  depict  this  iterative 
process  in  Figure  1. 
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Figure  1 :  The  Iterative  Nature  of  Experimentation,  illustrates  the  process  of  design, 
experimentation,  analysis,  and  redesign  (Box  and  Draper,  1987). 


RSM  follows  this  iterative  process  that  begins  with  fitting  a  first  order  plane  or  some 
other  multi-dimensional  model  using  orthogonal  design.  In  the  first  experiment,  or  screening 
experiment,  interaction  effects  are  considered  using  second  order  equations  and  conducting  an 
analysis  of  variance  (ANOVA)  to  determine  if  the  p-value  for  each  factor  is  significant  based  on 
a  given  confidence  interval.  If  the  factors  are  not  statistically  significant,  then  they  are  eliminated 
from  the  experimental  design.  The  next  step  is  to  find  the  path  of  steepest  ascent  or  descent  by 
conducting  single  or  replicated  runs  of  the  experiment  based  on  the  variables  in  the  design  for  the 
given  region  of  interest.  The  experimental  region,  including  the  fit  of  a  new  multidimensional 
model,  is  centered  on  a  point  along  the  path  of  steepest  ascent  or  descent  as  a  base  for  a  second 
experiment.  The  experiment  continues  in  this  process  until  response  indicates  a  near  optimum 
system  or  a  holistic  objective  has  been  achieved. 

Chapter  3:  Applied  Experiment 


3.1  Purpose 

The  purpose  of  this  example  experimental  design  is  to  demonstrate  the  usefulness  of  the 
methodology  and  the  feasibility  of  its  application  in  ballistic  experiments.  The  factors  in  this 
model  are  simulated  and  the  computer-generated  results  are  typical  of  those  used  by  scientists 
and  product  developers.  Ballistic  scientists  can  utilize  this  example  to  easily  tailor  an  experiment 
to  their  strategic  goals  given  their  complex  list  of  factors,  responses,  and  objectives. 
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3.2  Factor  Design 

In  order  to  demonstrate  the  usefulness  and  feasibility  of  this  methodology  we  designed  a 
simulated  experiment  analyzing  two  materials  of  ballistic  armor  for  use  in  Outer  Tactical  Vests 
(OTV).  These  vests  are  carriers  composed  of  soft  body  armor,  providing  protection  from  small 
arms  and  shrapnel.  The  OTVs  also  carry  Small  Arms  Protective  Inserts  (SAPI)  and  are  used  to 
affix  equipment  to  the  soldier’s  body.  We  developed  a  model  to  simulate  data  in  order  to  drive  a 
typical  experiment  that  can  show  usefulness  of  this  methodology.  The  simulated  data  both 
protects  the  sensitive  nature  of  the  actual  material  and  also  helps  to  better  communicate 
usefulness  of  the  methodology.  We  included  three  factors  which  are  material,  aramid  fiber  weave 
design,  and  areal  density.  These  particular  factors  are  representative  of  typical  considerations  for 
soft  body  armor  experimentation,  but  they  can  be  anything  of  importance  to  the  decision  maker. 
Designing  the  experiment  in  this  manner  will  compel  product  improvement  because  it  focuses  on 
differentiating  the  materials  through  type,  composition,  production  processes,  weight,  and  more. 
The  example  experimental  design  is  shown  in  Table  3. 

TABLE  3 

Applied  OTV  Experiment 
Level 

Factors  Low  (-)  High  (+) 

Material  Type  1  Type  2 

Areal  Density  1 .05  psf  1.10  psf 

Weave  Design  29x29  32x32 


3.3  Response 

It  is  important  to  understand  that  in  ballistic  experiments,  the  client  must  fire  a  large 
number  of  shots  to  determine  accurate  levels  of  performance.  As  with  anything  there  is  some 
amount  of  variability  in  the  sample.  Figure  2  illustrates  that  for  a  given  armor  material,  it  will 
perform  consistently  within  a  within  its  intended  range  of  use;  however,  soldiers  may  experience 
impacts  beyond  the  intended  level  of  threat  protection  in  the  zone  of  mixed  results.  In  this  graph, 
the  material  performs  consistently  well  at  a  given  velocity  VO  and  consistently  poor  at  a  velocity 
greater  than  VI 00. 
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Figure  2:  Statistical  Ballistic  Response,  illustrates  the  zone  of  mix  results  that  requires 
special  consideration  to  ensure  accurate  evaluation  of  armor  performance. 

Response  surface  methodology  will  allow  the  client  to  analyze  a  large  quantity  of  data  to 
determine  performance  within  the  zone  of  mixed  results  and  drive  future  experiments  to  improve 
the  system.  One  advantage  of  this  methodology  is  the  ability  to  analyze  a  binomial  response, 
whether  it  is  a  pass-fail  penetration  check  or  even  the  ability  to  determine  if  a  material  passed  a 
given  list  of  standards  beyond  just  penetration.  This  response  can  significantly  simplify  large 
amounts  of  data,  while  retaining  intelligent  design  features.  The  response  for  this  simulated 
experiment  is  a  binomial  variable  called  penetration. 

Chapter  4:  Results  of  the  Experiment 
4.1  Significant  Factors 

The  first  step  in  analysis  of  this  experiment  is  to  organize  the  data  into  an  orthogonal 
design  with  a  large  enough  sample  size  to  ensure  the  accuracy  of  response,  especially 
considering  its  statistical  nature.  Analysis  of  variance  shown  in  Table  4  identifies  factors  A  and 
B,  as  well  as  the  interaction  between  factor  A  and  B,  as  the  statistically  significant  factors  in  the 
experiment. 
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TABLE  4 


Analysis  of  Variance 


Df 

Sum  Sq 

Mean  Sq 

F  value 

Pr(>F) 

A 

1 

0.99488 

0.99488 

81226.15 

<  2e-16 

B 

1 

0.31841 

0.31841 

25996.6 

<  2e-16 

C 

1 

0.00005 

0.00005 

3.7396 

0.05348 

A:B 

1 

0.17091 

0.17091 

13953.81 

<2e-16 

A:C 

1 

0.00002 

0.00002 

1.594 

0.20711 

B:C 

1 

0.00001 

0.00001 

0.5035 

0.47818 

2.52E- 

A:B:C 

1 

06 

2.52E-06 

0.2058 

0.65017 

Residuals 

832 

0.01019 

0.00001 

In  this  experiment  factor  C,  the  aramid  fiber  weave  design,  does  not  affect  the  response 
and  can  be  eliminated  from  analysis.  The  significance  of  these  factors  and  their  interactions  can 
be  communicated  in  a  normal  plot  of  effects  as  illustrated  in  Figure  3.  Notice  the  deviation  of 
significant  factors  A,  B,  and  the  interaction  of  A  and  B. 
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Figure  3:  Normal  Plot  of  Effects,  illustrates  which  factors  are  significant. 

4.2  Response  Surface 

Once  the  design  is  complete  and  significance  is  determined,  a  response  for  the  significant 
factors  is  easily  computer-generated.  In  this  case,  a  three-dimensional  surface  is  required  to  show 
interactions  between  factors  A  and  B.  Another  experiment  may  require  multiple  surfaces  to 
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*  C 

*  A:C 
«  B:C 
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simplify  a  multivariable  problem  with  additional  significant  factors.  Unfortunately,  we  cannot 
illustrate  a  response  beyond  a  third  dimension.  The  response  surface  for  this  experiment  is  show 
in  Figure  4. 
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Figure  4:  Response  Surface  of  Body  Armor  illustrates  the  response  of  the  experiment  with 

respect  to  factors  A  and  B. 

4.3  Analysis  and  Application  of  the  Response  Surface 

This  experiment  illustrates  a  very  distinct  relationship  between  the  probability  of 
penetration  and  the  factors  material  type  and  areal  density.  This  is  a  minimization  problem, 
attempting  to  minimize  the  probability  of  penetration,  so  the  direction  of  steepest  descent  toward 
material  type  2,  the  binomial  high  value  of  Factor  A  in  the  experimental  design,  and  the  high 
value  of  areal  density. 

The  response  demonstrates  the  usefulness  of  the  methodology  because  it  illustrates  how 
clearly  the  decision  maker  can  determine  the  parameters  for  the  next  iteration  of 
experimentation.  This  response  indicates  the  next  experiment  should  include  material  more  like 
material  two  and  a  higher  areal  density  in  order  to  achieve  and  even  smaller  response.  However, 
this  experiment  also  indicates  the  high  values  of  the  factors  already  reached  zero  probability  of 
penetration,  so  no  further  experiment  is  required.  Had  the  lowest  probability  achieved  been 
greater  than  zero,  this  iterative  process  of  experimentation  could  continue  until  the  experiment 
yields  a  near  optimal  response. 
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In  this  case,  the  scientist  may  want  to  design  an  experiment  with  an  areal  density  between 
the  high  and  low  value  of  the  current  experiment  in  order  to  determine  the  response  of  lighter 
material.  This  next  experiment  would  be  an  effort  to  find  an  acceptable  probability  of  penetration 
at  the  lowest  weight,  a  tradeoff  extremely  important  to  body  armor  design. 

Chapter  5:  Conclusion 

This  experiment  demonstrates  the  usefulness  of  this  methodology  to  the  armor  industry, 
but  it  relies  on  the  expertise  of  armor  specialists  to  bridge  the  gap  between  data  simulation  and 
reality.  It  will  require  scientists  to  develop  similar  experimental  designs,  but  allow  the  flexibility 
of  this  methodology  to  encompass  the  factors  and  responses  inherent  to  their  unique  problems. 
The  design  will  drive  needed  coordination  between  scientists  and  producers  to  develop  the  many 
variations  the  armor  system  necessary  to  carry  out  the  experiment.  The  process  is  simple,  but  the 
results  are  far  reaching.  The  iterative  experimental  design  process  will  follow  a  direction  of 
intelligent  design,  and  the  statistical  nature  of  methodology  will  ensure  accuracy  of  the  response 
within  the  zone  of  mixed  results.  In  short,  this  improved  ballistic  test  and  evaluation 
methodology  can  result  in  a  better  product  and  safer  performance  evaluation.  It  is  ultimately 
beneficial  to  decision  makers,  product  developers,  and  of  course  the  soldiers  they  protect. 
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Appendix  A:  List  of  Abbreviations 


A 

ANOVA 

Analysis  of  Variance 

O 

OTV 

Outer  Tactical  Vest 

R 

RSM 

Response  Surface  Methodology 

S 

SAPI 

Small  Arms  Protective  Insert 

U 

USMA 

United  States  Military  Academy 

'his  table  is  sorted  al 

phabetically 
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